Committee Machines

In this chapter, we describe some of the most important architectures and algorithms for committee machines. We discuss three reasons for using committee machines. The first reason is that a committee can achieve a test set performance unobtainable by a single committee member. As typical representative approaches, we describe simple averaging, bagging and boosting. Secondly with committee machines, one obtains modular solutions which is sometimes desirable. The prime example here is the mixture of experts approach whose goal it is to autonomously break up a complex prediction task into subtasks which are modeled by the individual committee members. The third reason for using committee machines is a reduction in computational complexity. In the presented Bayesian committee machine, the training data set is partitioned into several smaller data sets and the different committee members are trained on the different sets. Their predictions are then combined using a covariance-based weighting scheme. The computational complexity of the Bayesian committee machine approach grows only linearly with the size of the training data set, independent of the learning systems used as committee members.